Transferring Learning from External to Internal Weights in Echo-State Networks with Sparse Connectivity
نویسندگان
چکیده
Modifying weights within a recurrent network to improve performance on a task has proven to be difficult. Echo-state networks in which modification is restricted to the weights of connections onto network outputs provide an easier alternative, but at the expense of modifying the typically sparse architecture of the network by including feedback from the output back into the network. We derive methods for using the values of the output weights from a trained echo-state network to set recurrent weights within the network. The result of this "transfer of learning" is a recurrent network that performs the task without requiring the output feedback present in the original network. We also discuss a hybrid version in which online learning is applied to both output and recurrent weights. Both approaches provide efficient ways of training recurrent networks to perform complex tasks. Through an analysis of the conditions required to make transfer of learning work, we define the concept of a "self-sensing" network state, and we compare and contrast this with compressed sensing.
منابع مشابه
Reservoir Computing for Sensory Prediction and Classification in Adaptive Agents
Artificial neural networks are an in silico laboratory for studying the dynamics of the brain. In recurrent networks, the units’ activations are recurrently fed back into the network. Thereby complex network dynamics emerge that extend over longer time scales than the individual units’ activation time constants. The recurrent echo-state networks with their fixed connection weights acquire an in...
متن کاملImproving the Prediction Accuracy of Echo State Neural Networks by Anti-Oja's Learning
Echo state neural networks, which are a special case of recurrent neural networks, are studied from the viewpoint of their learning ability, with a goal to achieve their greater prediction ability. A standard training of these neural networks uses pseudoinverse matrix for one-step learning of weights from hidden to output neurons. This regular adaptation of Echo State neural networks was optimi...
متن کاملMote-Based Online Anomaly Detection Using Echo State Networks
Sensor networks deployed for scientific data acquisition must inspect measurements for faults and events of interest. Doing so is crucial to ensure the relevance and correctness of the collected data. In this work we unify fault and event detection under a general anomaly detection framework. We use machine learning techniques to classify measurements that resemble a training set as normal and ...
متن کاملMerging Echo State and Feedforward Neural Networks for Time Series Forecasting
Echo state neural networks, which are a special case of recurrent neural networks, are studied from the viewpoint of their learning ability, with a goal to achieve their greater prediction ability. A standard training of these neural networks uses pseudoinverse matrix for one-step learning of weights from hidden to output neurons. Such learning was substituted by backpropagation of error learni...
متن کاملDistributed Sequence Memory of Multidimensional Inputs in Recurrent Networks
Recurrent neural networks (RNNs) have drawn interest from machine learning researchers because of their effectiveness at preserving past inputs for time-varying data processing tasks. To understand the success and limitations of RNNs, it is critical that we advance our analysis of their fundamental memory properties. We focus on echo state networks (ESNs), which are RNNs with simple memoryless ...
متن کامل